Skip to content

378 feature request make sure that results obtained from different benchmark versions are not compared#504

Open
dodu94 wants to merge 11 commits intodevelopingfrom
378-feature-request---make-sure-that-results-obtained-from-different-benchmark-versions-are-not-compared
Open

378 feature request make sure that results obtained from different benchmark versions are not compared#504
dodu94 wants to merge 11 commits intodevelopingfrom
378-feature-request---make-sure-that-results-obtained-from-different-benchmark-versions-are-not-compared

Conversation

@dodu94
Copy link
Copy Markdown
Member

@dodu94 dodu94 commented Mar 30, 2026

Fix #378

At last metadata are used to ensure that during post-processing of a specific benchmark (and code) the libraries results requested were all obtained from the same benchmark version.

To be reviewed after #500.

Summary by CodeRabbit

  • New Features

    • Added benchmark-version consistency validation to prevent processing mismatches and surface a clear error when versions differ.
    • Added ability to refresh parsed raw-results metadata on demand.
  • Refactor

    • Global status is now lazily initialized to improve startup behavior.
    • Simulation success detection now inspects run output file listings.
    • Raw-results metadata and simulation listings are parsed lazily and cached for reuse.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 30, 2026

Walkthrough

The PR lazy-loads GlobalStatus in JadeApp, moves raw-results and simulations parsing to cached properties, adds benchmark-version validation for requested libraries (raising VersionInconsistencyError on mismatch), refactors simulation success checkers into classes, updates run/post-processing to use the new checkers and validation, and adapts tests accordingly.

Changes

Cohort / File(s) Summary
App / Status Lazy Loading
src/jade/app/app.py, src/jade/config/status.py
JadeApp.status changed from an eagerly-set attribute to a lazy @property backed by _status. GlobalStatus no longer calls update() in init; simulations/raw_data/raw_metadata are lazy-cached properties.
Benchmark Version Validation
src/jade/config/status.py, src/jade/helper/errors.py
Added validate_codelibs() and _validate_libs_processing() to enforce consistent benchmark major versions across requested libs; new VersionInconsistencyError exception introduced.
Simulation Checker Refactor
src/jade/helper/aux_functions.py, src/jade/post/sim_output.py
Replaced procedural check functions with SimulationChecker base class and concrete checkers (MCNPChecker, OpenMCChecker, SerpentChecker, D1SChecker). CODE_CHECKERS now stores checker instances. Added is_successfully_simulated() helpers to MCNP/OpenMC outputs.
Run/Post-processing Integration
src/jade/run/benchmark.py, src/jade/app/app.py
Run continuation now calls CODE_CHECKERS[code].check_success(os.listdir(...)). post_process() now explicitly calls status.validate_codelibs(...) before creating outputs and running Excel/Atlas steps.
Tests & Fixtures
tests/app/test_app.py, tests/config/test_status.py, tests/dummy_structure/raw_data/_mcnp_-_ENDFB-VIII.0_/Oktavian/metadata.json
Tests adjusted to force lazy status evaluation and to call update_raw_results() where appropriate; added test for version inconsistency; updated dummy metadata benchmark_version from "1.0" → "2.0".

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant App as JadeApp
    participant Status as GlobalStatus
    participant Validator as _validate_libs_processing
    participant Post as Excel/Atlas Processing

    User->>App: post_process(codelibs, benchmark)
    App->>Status: access status (lazy-load if _status is None)
    Status-->>App: GlobalStatus instance
    App->>Status: validate_codelibs(codelibs, benchmark)
    Status->>Validator: _validate_libs_processing(code, benchmark, libs)
    alt Version mismatch
        Validator-->>Status: VersionInconsistencyError
        Status-->>App: raise error
        App-->>User: post_process fails
    else Versions consistent
        Validator-->>Status: OK
        Status-->>App: validation passed
        App->>Post: create outputs & run Excel/Atlas processing
        Post-->>App: processing complete
        App-->>User: post_process succeeds
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

  • C model doc update #432 — touches post-processing tally manipulation; may overlap conceptually with post-processing flow changes (verification recommended).

Suggested reviewers

  • alexvalentine94

Poem

🐰 I waited quietly, status tucked away,
Now checks hop in before we tally the day.
Classes of checkers paw through files with care,
Versions aligned — no mismatched affair!
🥕📊

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 54.05% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly corresponds to the main objective: preventing comparison of results from different benchmark versions.
Linked Issues check ✅ Passed The PR successfully implements the requested feature: version validation is enforced via the validate_codelibs() method that raises VersionInconsistencyError when benchmark versions differ, and post-processing explicitly calls this validation.
Out of Scope Changes check ✅ Passed All changes are directly related to implementing version consistency validation: lazy-loading optimization for status initialization, checker refactoring for compatibility, metadata loading, and the validation logic itself.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 378-feature-request---make-sure-that-results-obtained-from-different-benchmark-versions-are-not-compared

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 30, 2026

Codecov Report

❌ Patch coverage is 96.87500% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/jade/helper/aux_functions.py 88.88% 2 Missing ⚠️
src/jade/config/status.py 97.72% 1 Missing ⚠️
Files with missing lines Coverage Δ
src/jade/app/app.py 83.01% <100.00%> (+0.49%) ⬆️
src/jade/helper/errors.py 83.33% <100.00%> (+8.33%) ⬆️
src/jade/post/sim_output.py 93.56% <100.00%> (+0.13%) ⬆️
src/jade/run/benchmark.py 87.00% <100.00%> (ø)
src/jade/config/status.py 96.47% <97.72%> (+0.32%) ⬆️
src/jade/helper/aux_functions.py 95.18% <88.88%> (+1.27%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dodu94 dodu94 marked this pull request as ready for review March 30, 2026 14:17
@dodu94 dodu94 requested a review from alexvalentine94 March 30, 2026 14:17
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
tests/config/test_status.py (1)

52-60: Exercise the public validate_codelibs() path in this test.

JadeApp.post_process() calls the public wrapper with a [(CODE, lib), ...] list. Going straight to _validate_libs_processing() bypasses that grouping/dispatch layer, so this can miss bugs in the real production entry point and will break on harmless internal refactors.

♻️ Suggested test shape
-        with pytest.raises(VersionInconsistencyError):
-            status._validate_libs_processing(
-                CODE.MCNP, "Oktavian", ["FENDL 3.2c", "ENDFB-VIII.0"]
-            )
-        status._validate_libs_processing(
-            CODE.MCNP, "Sphere", ["FENDL 3.2c", "ENDFB-VIII.0"]
-        )
+        with pytest.raises(VersionInconsistencyError):
+            status.validate_codelibs(
+                [(CODE.MCNP, "FENDL 3.2c"), (CODE.MCNP, "ENDFB-VIII.0")],
+                "Oktavian",
+            )
+        status.validate_codelibs(
+            [(CODE.MCNP, "FENDL 3.2c"), (CODE.MCNP, "ENDFB-VIII.0")],
+            "Sphere",
+        )
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/config/test_status.py` around lines 52 - 60, The test should exercise
the public entrypoint validate_codelibs instead of calling the private helper
_validate_libs_processing directly; replace the two calls to
status._validate_libs_processing(...) with calls that pass a list of (CODE, lib)
pairs to status.validate_codelibs([(CODE.MCNP, "Oktavian")]) wrapped in
pytest.raises(VersionInconsistencyError) for the failing case, and
status.validate_codelibs([(CODE.MCNP, "Sphere")]) for the succeeding case so the
test follows the same grouping/dispatch path used by JadeApp.post_process().
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/jade/app/app.py`:
- Around line 58-69: The cached GlobalStatus returned by JadeApp.status can
become stale after write operations; update JadeApp so that methods that mutate
the simulations/raw trees (e.g., run_benchmarks, raw_process, and any other
writer methods) invalidate the cache by setting self._status = None after
completing their write work, and ensure any helper functions that modify the
tree do the same; locate the status property and add cache-clearing calls in the
implementations of run_benchmarks and raw_process (and post_process if it
mutates data) so subsequent accesses to JadeApp.status construct a fresh
GlobalStatus.

In `@src/jade/config/status.py`:
- Around line 52-55: The raw-data discovery currently requires metadata.json and
lets missing or malformed metadata raise KeyError later; update the logic in the
raw_data getter and in _parse_raw_results_folder to tolerate absent/malformed
metadata by returning None or an empty dict for self._raw_metadata when
metadata.json is missing or invalid, and change _validate_libs_processing to
check for the presence of keys like "benchmark_version" via .get or explicit
presence checks on self._raw_metadata before indexing so it raises a clear
ConsistencyError (or similar) with a descriptive message instead of a raw
KeyError; ensure raw_results_path, _raw_metadata, _raw_data,
_parse_raw_results_folder, and _validate_libs_processing are the symbols updated
so availability queries succeed and validation fails cleanly.
- Around line 46-48: The cached attribute self._simulations (accessed by the
simulations property) is never refreshed after first access, so methods like
was_simulated() and get_successful_simulations() can become stale; modify the
simulations property (and the similar block at lines ~66-72) to add an explicit
refresh path (e.g., an optional force_refresh flag or refresh method) that calls
self._parse_simulations_folder(self.simulations_path) to repopulate
self._simulations when update_raw_results() runs or when callers request a fresh
scan; update GlobalStatus to call that refresh (or pass force_refresh=True)
before was_simulated() / get_successful_simulations() use the cache so the
folder is rescanned for new/removed simulations.

In `@src/jade/helper/aux_functions.py`:
- Around line 72-126: The checkers lost run-directory context because
SimulationChecker.check_success(files: list[str]) only receives filenames;
change the abstract signature to check_success(files: list[str], run_dir: str |
Path) -> bool (or add an optional run_dir: str | Path = None) on
SimulationChecker and update all concrete implementations
(MCNPChecker.check_success, OpenMCChecker.check_success,
SerpentChecker.check_success, D1SChecker.check_success) to accept and use
run_dir when resolving/validating outputs (keeping existing behavior if run_dir
is None for backward compatibility); then update callers (e.g., the code paths
in src/jade/config/status.py and src/jade/run/benchmark.py) to pass the run
directory into check_success and update docstrings/type hints accordingly.

---

Nitpick comments:
In `@tests/config/test_status.py`:
- Around line 52-60: The test should exercise the public entrypoint
validate_codelibs instead of calling the private helper
_validate_libs_processing directly; replace the two calls to
status._validate_libs_processing(...) with calls that pass a list of (CODE, lib)
pairs to status.validate_codelibs([(CODE.MCNP, "Oktavian")]) wrapped in
pytest.raises(VersionInconsistencyError) for the failing case, and
status.validate_codelibs([(CODE.MCNP, "Sphere")]) for the succeeding case so the
test follows the same grouping/dispatch path used by JadeApp.post_process().
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: cf6c71ca-11de-4756-b1b1-8a13ced0ecf1

📥 Commits

Reviewing files that changed from the base of the PR and between ba664bd and c4c0ea4.

📒 Files selected for processing (9)
  • src/jade/app/app.py
  • src/jade/config/status.py
  • src/jade/helper/aux_functions.py
  • src/jade/helper/errors.py
  • src/jade/post/sim_output.py
  • src/jade/run/benchmark.py
  • tests/app/test_app.py
  • tests/config/test_status.py
  • tests/dummy_structure/raw_data/_mcnp_-_ENDFB-VIII.0_/Oktavian/metadata.json

Comment on lines +58 to +69
self._status = None

@property
def status(self) -> GlobalStatus:
"""Lazy-load the global status on first access."""
if self._status is None:
logging.info("Initializing the global status (lazy-loaded)")
self._status = GlobalStatus(
simulations_path=self.tree.simulations,
raw_results_path=self.tree.raw,
)
return self._status
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Invalidate the cached GlobalStatus after write operations.

This property memoizes one GlobalStatus snapshot for the lifetime of JadeApp. After run_benchmarks() or raw_process() changes the simulations/raw trees, later calls still consult stale simulations / raw_data unless callers refresh manually — tests/app/test_app.py Lines 63-72 now has to do that explicitly. A same-session raw_process()post_process() flow can therefore miss the raw results it just produced.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/jade/app/app.py` around lines 58 - 69, The cached GlobalStatus returned
by JadeApp.status can become stale after write operations; update JadeApp so
that methods that mutate the simulations/raw trees (e.g., run_benchmarks,
raw_process, and any other writer methods) invalidate the cache by setting
self._status = None after completing their write work, and ensure any helper
functions that modify the tree do the same; locate the status property and add
cache-clearing calls in the implementations of run_benchmarks and raw_process
(and post_process if it mutates data) so subsequent accesses to JadeApp.status
construct a fresh GlobalStatus.

Comment on lines +46 to +48
if self._simulations is None:
self._simulations = self._parse_simulations_folder(self.simulations_path)
return self._simulations
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add a refresh path for cached simulation state.

After the first simulations access, this instance never rescans simulations_path. update_raw_results() only refreshes raw outputs, so was_simulated() and get_successful_simulations() can stay stale for the lifetime of a long-lived GlobalStatus.

Possible fix
+    def update_simulations(self) -> None:
+        """Update the simulations by re-parsing the simulations folder."""
+        self._simulations = self._parse_simulations_folder(self.simulations_path)
+
     def update_raw_results(self) -> None:
         """Update the raw results by re-parsing the raw results folder. It should be used
         after processing new raw results to update the status.
         """
         self._raw_data, self._raw_metadata = self._parse_raw_results_folder(
             self.raw_results_path
         )

Also applies to: 66-72

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/jade/config/status.py` around lines 46 - 48, The cached attribute
self._simulations (accessed by the simulations property) is never refreshed
after first access, so methods like was_simulated() and
get_successful_simulations() can become stale; modify the simulations property
(and the similar block at lines ~66-72) to add an explicit refresh path (e.g.,
an optional force_refresh flag or refresh method) that calls
self._parse_simulations_folder(self.simulations_path) to repopulate
self._simulations when update_raw_results() runs or when callers request a fresh
scan; update GlobalStatus to call that refresh (or pass force_refresh=True)
before was_simulated() / get_successful_simulations() use the cache so the
folder is rescanned for new/removed simulations.

Comment on lines +52 to +55
if self._raw_data is None:
self._raw_data, self._raw_metadata = self._parse_raw_results_folder(
self.raw_results_path
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Make raw-data discovery tolerant to missing metadata, then fail cleanly during validation.

raw_data now depends on metadata.json being present for every non-exp benchmark, so folders without that file will break simple availability queries. _validate_libs_processing() then indexes "benchmark_version" directly, which turns malformed metadata into a raw KeyError instead of a clear consistency error.

Possible fix
-                available_raw_data[(CODE(code), lib, benchmark)] = os.listdir(
-                    bench_path
-                )
+                available_raw_data[(CODE(code), lib, benchmark)] = [
+                    entry for entry in os.listdir(bench_path)
+                    if entry != "metadata.json"
+                ]
                 if lib != "exp":  # for the experiments we don't have metadata
-                    with open(os.path.join(bench_path, "metadata.json")) as infile:
-                        metadata[(CODE(code), lib, benchmark)] = json.load(infile)
+                    metadata_file = Path(bench_path, "metadata.json")
+                    if metadata_file.is_file():
+                        with metadata_file.open() as infile:
+                            metadata[(CODE(code), lib, benchmark)] = json.load(infile)
...
         versions = {}
         for lib in libs:
-            versions[lib] = self.raw_metadata[(code, lib, benchmark)][
-                "benchmark_version"
-            ]
+            version = self.raw_metadata.get((code, lib, benchmark), {}).get(
+                "benchmark_version"
+            )
+            if version is None:
+                raise VersionInconsistencyError(
+                    f"Missing benchmark_version metadata for {code} / {lib} / {benchmark}"
+                )
+            versions[lib] = version

Also applies to: 60-63, 144-146, 292-294

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/jade/config/status.py` around lines 52 - 55, The raw-data discovery
currently requires metadata.json and lets missing or malformed metadata raise
KeyError later; update the logic in the raw_data getter and in
_parse_raw_results_folder to tolerate absent/malformed metadata by returning
None or an empty dict for self._raw_metadata when metadata.json is missing or
invalid, and change _validate_libs_processing to check for the presence of keys
like "benchmark_version" via .get or explicit presence checks on
self._raw_metadata before indexing so it raises a clear ConsistencyError (or
similar) with a descriptive message instead of a raw KeyError; ensure
raw_results_path, _raw_metadata, _raw_data, _parse_raw_results_folder, and
_validate_libs_processing are the symbols updated so availability queries
succeed and validation fails cleanly.

Comment on lines +72 to +126
class SimulationChecker(ABC):
"""Abstract base class for simulation success checkers.

All code-specific checkers must inherit from this class and implement
the check_success method with the same signature.
"""

def check_run_openmc(folder: PathLike) -> bool:
"""check if openmc run was successful"""
try:
OpenMCSimOutput.retrieve_file(folder)
return True
except FileNotFoundError:
return False
@abstractmethod
def check_success(self, files: list[str]) -> bool:
"""Check if a simulation run was successful.

Parameters
----------
files : list[str]
List of output files to check for success.

Returns
-------
bool
True if the simulation completed successfully, False otherwise.
"""
pass


class MCNPChecker(SimulationChecker):
"""Checker for MCNP simulations."""

def check_success(self, files: list[str]) -> bool:
"""Check if MCNP run was successful by verifying output files exist."""
return MCNPSimOutput.is_successfully_simulated(files)


class OpenMCChecker(SimulationChecker):
"""Checker for OpenMC simulations."""

def check_success(self, files: list[str]) -> bool:
"""Check if OpenMC run was successful by verifying output files exist."""
return OpenMCSimOutput.is_successfully_simulated(files)


class SerpentChecker(SimulationChecker):
"""Checker for Serpent simulations."""

def check_success(self, files: list[str]) -> bool:
"""Check if Serpent run was successful."""
# TODO implement the logic to check if the Serpent run was successful
raise NotImplementedError("Serpent checker not yet implemented")

def check_run_serpent(folder: PathLike) -> bool:
# TODO implement the logic to check if the Serpent run was successful
raise NotImplementedError

class D1SChecker(SimulationChecker):
"""Checker for D1S simulations."""

def check_run_d1s(folder: PathLike) -> bool:
"""check if d1s run was successful"""
return check_run_mcnp(folder)
def check_success(self, files: list[str]) -> bool:
"""Check if D1S run was successful (uses same logic as MCNP)."""
return MCNPChecker().check_success(files)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Keep the run directory in check_success().

The new files: list[str] contract strips the context needed to validate actual completion. As used by src/jade/config/status.py Lines 94-98 and src/jade/run/benchmark.py Lines 441-445, these checkers now have to infer success from filename presence alone, which can mark interrupted runs with leftover artifacts as complete and then skip reruns or post-process bad outputs.

💡 Suggested interface direction
 class SimulationChecker(ABC):
     """Abstract base class for simulation success checkers.
@@
     `@abstractmethod`
-    def check_success(self, files: list[str]) -> bool:
+    def check_success(
+        self, run_dir: PathLike, files: list[str] | None = None
+    ) -> bool:
         """Check if a simulation run was successful.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/jade/helper/aux_functions.py` around lines 72 - 126, The checkers lost
run-directory context because SimulationChecker.check_success(files: list[str])
only receives filenames; change the abstract signature to check_success(files:
list[str], run_dir: str | Path) -> bool (or add an optional run_dir: str | Path
= None) on SimulationChecker and update all concrete implementations
(MCNPChecker.check_success, OpenMCChecker.check_success,
SerpentChecker.check_success, D1SChecker.check_success) to accept and use
run_dir when resolving/validating outputs (keeping existing behavior if run_dir
is None for backward compatibility); then update callers (e.g., the code paths
in src/jade/config/status.py and src/jade/run/benchmark.py) to pass the run
directory into check_success and update docstrings/type hints accordingly.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
src/jade/config/status.py (1)

314-321: Redundant CODE conversion.

Line 316 converts code to CODE(code), but the type hint on line 302 declares codelibs: list[tuple[CODE, str]], indicating code should already be a CODE enum. If callers may pass strings, consider updating the type hint; otherwise, remove the redundant conversion.

♻️ Proposed simplification (if type hint is accurate)
         for code, lib in codelibs:
-            code = CODE(code)
             if lib == "exp":  # for the experiments we don't have metadata
                 continue
             if code not in libs:
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/jade/config/status.py` around lines 314 - 321, The loop is redundantly
converting code to CODE(code) even though codelibs is typed as list[tuple[CODE,
str]]; remove the unnecessary conversion in the for loop (i.e., stop reassigning
code = CODE(code)) so the existing CODE instance from codelibs is used directly,
or if callers may pass strings instead of CODE update the codelibs type hint and
add validation/conversion at the call sites; target the for loop that iterates
over codelibs and the use of CODE to decide which path to take.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/jade/config/status.py`:
- Around line 301-312: The validate_codelibs method's parameter is named
benchmarks but its type hint and docstring describe a single benchmark; rename
the parameter to benchmark to match the docstring and type hint and update any
internal uses and callers accordingly (search for validate_codelibs and change
the parameter name in its signature and all invocations to benchmark) so
documentation and code are consistent.
- Around line 285-288: The docstring is incorrect: update the docstring for the
function that raises VersionInconsistencyError (the function documented in this
block) to state that it returns None on success and to include a Raises section
documenting VersionInconsistencyError instead of claiming it returns bool;
specifically change the Returns section to "None" (or remove it) and add a
"Raises\n-------\nVersionInconsistencyError: when post-processing cannot be
performed" so the docstring matches the actual behavior.

---

Nitpick comments:
In `@src/jade/config/status.py`:
- Around line 314-321: The loop is redundantly converting code to CODE(code)
even though codelibs is typed as list[tuple[CODE, str]]; remove the unnecessary
conversion in the for loop (i.e., stop reassigning code = CODE(code)) so the
existing CODE instance from codelibs is used directly, or if callers may pass
strings instead of CODE update the codelibs type hint and add
validation/conversion at the call sites; target the for loop that iterates over
codelibs and the use of CODE to decide which path to take.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 61633b82-8572-41db-b231-8cba6472f1fe

📥 Commits

Reviewing files that changed from the base of the PR and between c4c0ea4 and 1b8e7f9.

📒 Files selected for processing (1)
  • src/jade/config/status.py

Comment on lines +285 to +288
Returns
-------
bool
True if the post-processing can be performed, False otherwise.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Docstring incorrectly states return type.

The docstring states "Returns -> bool: True if the post-processing can be performed" but the method returns None or raises VersionInconsistencyError. Update the docstring to reflect the actual behavior.

📝 Proposed docstring fix
-        Returns
-        -------
-        bool
-            True if the post-processing can be performed, False otherwise.
+        Raises
+        ------
+        VersionInconsistencyError
+            If the benchmark versions are not consistent across the requested libraries.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/jade/config/status.py` around lines 285 - 288, The docstring is
incorrect: update the docstring for the function that raises
VersionInconsistencyError (the function documented in this block) to state that
it returns None on success and to include a Raises section documenting
VersionInconsistencyError instead of claiming it returns bool; specifically
change the Returns section to "None" (or remove it) and add a
"Raises\n-------\nVersionInconsistencyError: when post-processing cannot be
performed" so the docstring matches the actual behavior.

Comment on lines +301 to +312
def validate_codelibs(
self, codelibs: list[tuple[CODE, str]], benchmarks: str
) -> None:
"""Check that a list of codelibs can be post-processed together for a given benchmark.
This is true if the benchmark version for the requested libs are the same.

Parameters
----------
codelibs : list[tuple[CODE, str]]
list of codelibs to check. Each codelib is a tuple with the code and library.
benchmark : str
benchmark name.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Parameter name and docstring inconsistency.

The parameter is named benchmarks (line 302) but the docstring references benchmark (line 311-312). Since the type hint indicates str (singular benchmark), consider renaming the parameter to benchmark for consistency.

📝 Proposed fix
     def validate_codelibs(
-        self, codelibs: list[tuple[CODE, str]], benchmarks: str
+        self, codelibs: list[tuple[CODE, str]], benchmark: str
     ) -> None:
         """Check that a list of codelibs can be post-processed together for a given benchmark.
...
         for code, lib_list in libs.items():
-            self._validate_libs_processing(code, benchmarks, lib_list)
+            self._validate_libs_processing(code, benchmark, lib_list)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/jade/config/status.py` around lines 301 - 312, The validate_codelibs
method's parameter is named benchmarks but its type hint and docstring describe
a single benchmark; rename the parameter to benchmark to match the docstring and
type hint and update any internal uses and callers accordingly (search for
validate_codelibs and change the parameter name in its signature and all
invocations to benchmark) so documentation and code are consistent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature request] - make sure that results obtained from different benchmark versions are not compared

1 participant